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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
APPLICATION FOR PATENT 

Title: TWO STAGE DETECTION FOR PHOTOGRAPHIC EYE ARTIFACTS 

5 

Inventors: Yury Prilutsky, Alexei Pososin, Mihai Ciuc, Petronel Bigioi, Eran Steinberg 
and Peter Corcoran 

Atty. Docket: FN-121-CIP3-PCT 

10 

PRIORITY AND RELATED APPLICATIONS 

This application claims the benefit of priority to United States provisional patent 
application no. 60/892,884, filed March 5, 2007, which is incorporated by reference. This 
15 application is also a Continuation-in-Part (CIP) of United States patent applications nos. 

1 1/462,035, filed August 2, 2006 and 1 1/282,954, filed November 18, 2005 and published as 
US2007/01 16379, which are incorporated by reference. 

FIELD OF THE INVENTION 
20 The present invention relates to digital image processing, and more particularly to a 

method and apparatus for detection and correction of red-eye defects and other artifacts in an 
acquired digital image. 

BACKGROUND 

25 Redeye is the appearance of an unnatural reddish coloration of the pupils of a person 

appearing in an image captured by a camera with flash illumination. Redeye is caused by light 
from the flash reflecting off blood vessels in the person's retina and returning to the camera. 

A large number of image processing techniques have been proposed to detect and 
correct redeye in color images. In general, these techniques typically are semi-automatic or 
30 automatic. Semi-automatic redeye detection techniques rely on human input. For example, in 
some semi-automatic redeye reduction systems, a user must manually identify to the system 
the areas of an image containing redeye before the defects can be corrected. 

Many automatic redeye reduction systems rely on a preliminary face detection step 
before redeye areas are detected. A common automatic approach involves detecting faces in 
35 an image and, subsequently, detecting eyes within each detected face. After the eyes are 
located, redeye is identified based on shape, coloration, and brightness of image areas 
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corresponding to the detected eye locations. In general, face-detection-based automatic redeye 
reduction techniques have high computation and memory resource requirements. In addition, 
most of the face detection algorithms are only able to detect faces that are oriented in an 
upright frontal view; these approaches cannot detect faces that are rotated in-plane or out-of- 
5 plane with respect to the image plane. 

A typical prior art redeye filter process is illustrated in Fig 1(a). An input image is first 
analyzed by a speed optimized redeye detection stage 100 at a pixel level 103 and segmented 
into candidate redeye regions 104. A further series of falsing and verification filters 106 are 
then applied to the candidate regions and a set of confirmed redeye regions 108 is thus 
10 determined. A correction filter (pixel modifier) 102 is next applied to the confirmed regions 
and a final image 112, corrected for redeye, is generated. 

Exemplary prior art includes US patent 6,407,777 to DeLuca which discloses in- 
camera detection and correction of redeye pixels in an acquired digital image; US patent 
application 2002/0176623 to Steinberg which discloses automated real-time detection and 

15 correction of redeye defects optimized for handheld devices; US patent applications 
2005/0047655 and 2005/0047656 to Luo et al which disclose detecting and correcting redeye 
in a digital image and in embedded systems respectively. 

Now it is well known that within an image acquisition subsystem such as is embodied 
in typical digital cameras, the peak computing load and resource requirements occur around 

20 the time of image acquisition. Upon receiving an image acquisition request from the user the 
main embedded processing system must refine the image focus and exposure to achieve an 
optimal main acquired image; this image, in turn, must be off-loaded from the main optical 
sensor of the camera and subjected to further image processing to convert it from its raw 
format (e.g. Bayer) to a conventional color space such as RGB or YCC. Finally the acquired 

25 image must be compressed prior to saving it on a removable storage medium such as a 
compact flash or multimedia card. 

The time taken by the camera to recover from the acquisition of a first image and 
reinitialize itself to capture a second image is known in the industry as the "click-to -click" 
time. As this is one of the most important parameters for the comparison and marketing of 

30 modern digital cameras it vital for manufacturers to minimize said "click-to-click" time. Thus 
any additional image processing, such as redeye filtering, which is to be added to the main 
image acquisition chain should be highly optimized for speed of execution in order to 
minimize its impact on the click-to-click time of the main system. 
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Evidently such a redeye filter must compromise its overall performance in terms of 
accuracy of detection of redeye defects and quality of image correction. An alternative would 
be to wait until after the main image has been acquired and perform the redeye filtering at a 
later time when the camera may execute the filter as a background process, or to perform the 
5 redeye filtering off- camera on a desktop PC or printer. 

However there are some drawbacks to this approach. Firstly, images will be displayed 
on the acquiring device, immediately after acquisition, with uncorrected redeye defects; and, 
when images are accessed in playback mode, there will be a further delay while images are 
post-processed before an image can be displayed. Both drawbacks would create a negative 
1 0 impression on end users. 

Further, as practically all digital cameras store images using lossy compression 
techniques there are additional disadvantages with respect to image quality as images must be 
decompressed and recompressed in order to perform the redeye detection and correction 
processes in playback or background modes. Such loss of image quality may not become 
15 apparent until later when a user wishes to print an image and it is too late to reverse the 
process. 

If redeye processing is delayed until the images are loaded onto another device, such as 
a desktop PC or printer there are further disadvantages. Firstly, important meta-data relating 
to the acquiring device and its state at the time the image was acquired may not be available 
20 to the redeye filter process. A second disadvantage is that this post-processing device must 
perform redeye filtering on the entire image; where this is an embedded device such as a 
printer it may, itself, be relatively constrained in terms of CPU cycles and processing 
resources for its primary post-processing activity and it may be desirable to optimize the 
performance of the full redeye filter. 

25 

SUMMARY OF THE INVENTION 
The detection of red-eye defects is enhanced in digital images for embedded image 
acquisition and processing systems. A two-stage redeye filtering process is provided whereby 
a speed optimized filter performs the initial segmentation of candidate redeye regions and 
30 optionally applies a speed- optimized set of falsing/verification filters to determine a first set 
of confirmed redeye regions for correction. Some of the candidate regions which are rejected 
during the first stage are recorded and re-analyzed during a second stage by an alternative set 
of analysis-optimized filters to determine a second set of confirmed redeye regions. 
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In another embodiment, the first set of confirmed redeye regions may be passed 
through the stage-two analysis- optimized filters. 

In an another embodiment the second stage filter may incorporate an enhanced 
correction filter which may be optionally applied to the first set of confirmed redeye regions. 
5 A two-stage redeye filter is implemented wherein a first redeye filter process, which is 

optimized for speed is combined with a second redeye process which is optimized for 
accurate image analysis. Thus the advantages of a redeye filter implemented within the main 
image acquisition chain are combined with those of a redeye filter implemented in 
background/playback mode and the disadvantages of these two approaches are mostly 
10 eliminated. 

For slow platforms (like Fujitsu for Samsung), in playback mode, a "RE lib" is called 
after, and preferably immediately after, an image is displayed on the screen (e.g., without 
waiting for the user's command). The camera is capable of multi-tasking and/or can quickly 
abandon a background task when there is a need to do something else. This gives a user an 

15 instantaneous or real-time result when the RE function is executed. The presence of RE may 
be provided in a picture, e.g., in a flashing region. 

A farther technique is provided for processing a digital image. Candidate face regions 
are obtained of an acquired digital image. The candidate face regions are filtered with a first 
speed optimized filter to produce a first set of candidate red-eye regions. At least a portion of 

20 the acquired digital image is encoded. The encoded portion of the acquired digital image is 
stored in association with said first set of candidate red-eye regions for later image processing 
of said encoded image. 

A speed-optimized set of one or more falsing/verification filters may also be applied. 
The first set of candidate redeye regions may be passed through an analysis-optimized filter. 

25 Candidate face regions rejected by the filtering may be passed through an analysis-optimized 
filter sometime after the filtering, encoding and storing, such that one or more may be 
determined as candidate red-eye regions. The first speed optimized filter may be applied 
upon image acquisition, and the analysis-optimized filter may be applied in a background or 
playback mode or both. 

30 This is applicable to more complex processes such as Golden Eye removal (e.g., a 

"two stage" process may be removing red-eye in real time and Golden Eye in the background. 
A two-stage process may include a fast filter during an acquisition chain and a slower, 
analysis optimized filter applied afterwards, when the camera is idling, or even on a 
secondary device. This helps particularly slow machines, and can also be not only predictive 
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for playback but also a background process when a camera is not taking pictures in general. 
There can be an icon that shows that Red Eye was performed and/or a flashing region on a 
display. 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1(a) illustrates a typical prior art redeye process; 
Fig. 1(b) illustrates a redeye process according to an embodiment; 
Fig. 1(c) illustrates a redeye process according to an alternative embodiment; 
10 Fig. 2(a) illustrates an embodiment of the present invention within a digital image 

acquisition device; 

Fig. 2(b) illustrates an embodiment of the present invention wherein the analysis 
optimized redeye filter is performed on a separate device to the original acquiring device; 

Fig. 3(a) illustrates a process according to an embodiment of the present invention 
15 whereby the speed optimized redeye detector is applied to a partially compressed DCT block 
image; 

Fig. 3(b) is a workflow diagram of an illustrative embodiment of an improved in- 
camera redeye detection means employing a redeye DCT prefilter; 

Fig. 3(c) is a workflow diagram of an illustrative embodiment of the redeye DCT 
20 prefilter; 

Fig. 3(d) illustrates a segmentation step of the redeye DCT prefilter; 

Fig. 3(e) shows a 4-DCT block neighborhood; 

Fig. 4(a) illustrates eye regions mapped onto a rectangular grid; 

Fig. 4(b) illustrates the approximate color which will be recorded by the DC 
25 coefficient of each DCT block after the image of Fig 4(a) is transformed into the DCT 
domain; 

Figs 4(c), 4(d) and 4(e) illustrate the DCT blocks from Fig 4(a) which can be 
identified with the colors of a redeye candidate region, an eye-white region and a skin color 
region, respectively, through the use of an inclusive color determining filter method; 
30 Fig. 5 illustrates a functional implementation of modified redeye filtering process 

according to an embodiment; 

Fig 6(a) illustrates an example of the original defect region stored in a header and a 
corrected defect region applied to a main image body; 
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Fig 6(b) illustrates an example of the corrected defect region stored in the header and 
the original defect region remaining xmcorrected in the main image body; 

Fig 6(c) illustrates an example of the original defect region and at least one alternative 
corrected defect region stored in the header and the optimally determined corrected defect 
region applied to the main image body; and 

Figs 7 and 8 illustrate functional implementations of modified redeye filtering 
processes according to further embodiments. 
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DETAILED DESCRIPTION OF THE EMBODIMENTS 
The detection of red-eye defects is enhanced in digital images for embedded image 
acquisition and processing systems. A two-stage redeye filtering process is provided whereby 
a speed optimized filter performs the initial segmentation of candidate redeye regions and 
5 optionally applies a speed- optimized set of falsing/verification filters to determine a first set 
of confirmed redeye regions for correction. Some of the candidate regions which are rejected 
during the first stage are recorded and re-analyzed during a second stage by an alternative set 
of analysis-optimized filters to determine a second set of confirmed redeye regions. 

In another embodiment, the first set of confirmed redeye regions may be passed 
10 through the stage-two analysis-optimized filters. 

In an another embodiment the second stage filter may incorporate an enhanced 
correction filter which may be optionally applied to the first set of confirmed redeye regions. 

A two-stage redeye filter is implemented wherein a first redeye filter process, which is 
optimized for speed is combined with a second redeye process which is optimized for 
15 accurate image analysis. Thus the advantages of a redeye filter implemented within the main 
image acquisition chain are combined with those of a redeye filter implemented in 
background/playback mode and the disadvantages of these two approaches are mostly 
eliminated. 

One generalized embodiment is illustrated in Fig 1(b). An input image 110 is 
20 processed by a pixel analyzer 103, segmented into a set of candidate regions 104 and 
subsequently passed through a set of falsing & verification filters 106. All of these 
components form a speed optimized redeye detection filter 100 corresponding generally to the 
filter 1 00 of Figure 1 (a), except that in the embodiment the filter 1 00 is modified so that 
candidate redeye regions which, in the prior art speed optimised redeye filter, would have 
25 been ultimately classified as false positives, based on their size or probability being below a 
predetermined threshold are, instead saved as candidate regions 109 for a subsequent 
optimized analysis 101. 

Thus, in this embodiment the falsing & verification filters 106 generates a set of 
secondary candidate regions 109 in addition to the set of confirmed redeye regions 108. The 
30 set of secondary candidate regions may include members of the original candidate region set 
104, which could be neither confirmed nor eliminated by the speed optimized redeye 
detection process 100. It may also include combined candidate regions in close proximity to 
each other. 
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This set of candidate regions 109 is saved either in a RAM buffer, or in non- volatile 
memory depending on the implementation of the embodiment. Where the data is saved in 
RAM (or volatile) memory, the image acquisition system must apply the second stage redeye 
filter to the image prior to powering down. The preferred form of storage is in non-volatile 
5 memory, or on a removable media card. In other embodiments this data may be stored in the 
image header with the part-processed image itself. 

In this embodiment, a second stage, analysis optimised redeye filter 101 is next 
applied to the secondary set of candidate regions 109. During the analysis optimized detection 
process 101, the saved candidate regions 109 are preferably farther analyzed at a higher 

10 resolution than during the speed optimized process. Thus, the filter 101 includes an analysis 
optimized set of falsing and verification filters 116, which differ either in their nature or in 
their operating parameters from the falsing and verification filters 1 06 employed in the speed 
optimized analysis. Nonetheless, it will be appreciated that it may be useful to perform one or 
more intermediate stages of optimized analysis at increasing image resolutions. This will 

15 depend on the hardware capabilities of the imaging appliance and the resources available 
within the image processing subsystems of the imaging appliance. 

Second stage analysis may occur in response to a variety of external events. For 
example, a user may initiate image playback causing this filter 101 to be applied. 
Alternatively, a camera may signal that it has been idle for a predetermined interval and thus 

20 background redeye processing may be initiated. Where a camera can determine its motion, for 
example, from auto-focus data, it may be assumed that when a camera is idle, for example, 
where image focus does not change for a predetermined interval and no user input is received, 
background image processing, including stage-two redeye filtering, may be initiated. 

After a set of confirmed redeye regions 1 1 8 is determined by the second stage redeye 

25 filter 101, a correction filter (pixel modifier) 102 is applied and these corrected regions are 
merged 115 with the initial corrected image 1 12 to generate a final corrected image 113. 

An alternative embodiment is illustrated in Fig 1(c) which differs from the 
embodiment of Fig 1(b) in that a single correction filter (pixel modifier) 102b is applied after 
the second stage redeye filter 101, rather than merging the initial corrected image 112 with 
30 the corrected regions determined by the stage-two filter 101. The filter 102b corrects both the 
original confirmed redeye regions 108 and the second stage confirmed redeye regions 1 18 to 
produce the final corrected image 113. 
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Fig 2(a) illustrates an embodiment of the present invention within a digital image 
acquisition device. The speed optimized redeye filter 411 may contains both detection 411-1, 
411-2 & 411-4 and, optionally, correction 411-3 processes. Similarly, the analysis optimized 
redeye filter 412, which may operate as a background process 403, performs additional 
5 refinements to the initial determinations and corrections of the speed optimized filter 411. 
Data related to these initial determinations is provided by the redeye filter metadata 410-5 
which is stored with the acquired image 410-2 in an image store 410. 

Fig 2(b) illustrates a variation on the embodiment of Fig 2(a) wherein the analysis 
optimized redeye filter is performed on a separate device 400 to the original acquiring device. 

10 This may be, for example, a desktop PC, or a printer. In other embodiments the camera may 
connect directly to a network or web service. The image data transfer means 404a, 404b may 
be either a point-to-point communications link between the two devices; a removable storage 
media which is physically exchanged between the two devices, or alternatively both devices 
may be connected to a common network such as the internet. In other embodiments the 

15 redeye filter metadata 410-5 may be incorporated with the main image data 410-2 by adding 
the metadata to the JPEG header, see Figure 6. It will also be evident that background redeye 
filters may operate on both the original acquiring device 400 and the separate device 400'. 
We remark, however, that supporting multiple redeye filters of increasing sophistication 
requires very complex and detailed metadata to be exchanged and stored with the image being 

20 analyzed and corrected. 

For an exemplary, non- exhaustive list of some typical filters that may be found in 
either the speed or analysis-optimized redeye filters 411, 412, see US Application No. 
10/976,336 (Reference FN206US). 

In the embodiments above, the speed optimized redeye detection 100 is preferably 

25 applied to a sub-sampled input image. The confirmed redeye regions 108 from this speed 
optimized redeye detection 100 are passed to a redeye correction module 102/ 102a. The 
corrected redeye image 112 can be displayed on a low-resolution viewing screen of a digital 
camera immediately after the image acquisition process providing the user with a redeye 
corrected image almost instantly. However, although this initial corrected image 112 may be 

30 adequately corrected, for example, where it is a portrait- style image in which a face occupies 
most of an image or where large high probability red-eye regions exist, it may not be 
adequately corrected for images including a large groups of persons, where the candidate 
redeye regions are smaller or less certain. Accordingly, the second analysis optimized redeye 
filtering process 101 is implemented after image acquisition but prior to final image 113 
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display on a larger viewer, or image printing. We remark that one advantage of the present 
invention is that the analysis optimized redeye detection 101 and correction 102 processes 
may be delayed until such high resolution viewing or printing is desired by the end user. 

It should be noted that in the embodiments of Figures 2(a) and 2(b), the sub-sampled 
5 versions of the main image or well as uncorrected fall size versions of the main image may be 
provided directly from main image acquisition device hardware 402 rather than needing to 
explicitly sub-sample a decoded full size main image. 

As in the case, of Figure 1(c), it will be seen that image correction need not be 
performed on images within the acquisition chain and can in fact be performed in the 

10 background on acquired images for which speed optimised redeye detection has been 
performed in the acquisition chain. This is advantageous in many image acquisition 
appliances where image compression is often implemented in hardware as part of the main 
image acquisition chain 401. In this embodiment, only the detection process is actually 
performed in the acquisition chain. A speed optimized correction or a full analysis optimized 

15 redeye filter may be subsequently selected in playback mode either based on a predetermined 
setting within the camera, or on a user selection at the time of image playback/ vie wing. 

In the embodiment of Fig 3(a), an acquired raw image 402 is partially processed 404 
before being provided to DCT compression block 408-1. This block essentially provides a 
sub-sampled version of the acquired image and, although not shown, this can be provided to 

20 the image store 410 as explained above. A speed optimized redeye detector 428 is then 
applied to the partially compressed DCT block image and DCT red-eye candidate regions 
both corrected and suspected uncorrected regions are output for storage in the store 410. An 
advantage of applying speed optimised correction at DCT block level, rather than at the pixel 
level, is that the need for an explicit image subsampling step is avoided, yet the benefits of 

25 applying redeye analysis to a sub-sampled image as detailed in the prior art cited herein are 
retained. 

The regions output by the DCT prefilter 428, incorporated in the main image 
acquisition chain 401, can advantageously allow much of the DCT block stream to be 
bypassed without being processed when an image is subsequently corrected by a filter such as 
30 a background filter module 426. This allows either much faster or more detailed analysis and 
filtering of the DCT blocks which are determined to require processing by an analysis 
optimized redeye filter 406. Those skilled in the art will realize that further embodiments are 
possible which separate aspects of both the DCT prefilter and the otherwise conventional type 
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redeye filter 406 between the main image acquisition chain, 401 and a background redeye 
detection and correction process 426. 

Fig. 3(b) shows in more detail the operation of the redeye DCT prefilter 428. This 
particular example illustrates how the DCT prefilter can integrate with the main image 
5 acquisition, processing and compression chain, 402, 404 and 408 of Fig 3(a). The DCT image 
to be filtered is first loaded into memory 902 after which the main DCT prefilter 428 is 
applied. This has three main steps: firstly the DCT blocks of the image are scanned 904 and 
the relevant DCT coefficients are extracted. Depending on the sophistication of the filter, only 
the DC components of each DCT block may be utilized in the subsequent analysis. 

1 0 Alternatively, some of the AC components may be extracted in order to allow some texture or 
sharpness/blur determination as part of the prefilter operation. 

In the second principle stage of the DCT prefilter the DCT blocks are segmented and 
grouped 906 based on a plurality of criteria determined from the coefficients extracted at step 
904. Finally a region based analysis is performed 907 in order to determine the final candidate 

15 redeye groupings. Next it is determined if there are any valid candidate grouping 908 and if 
not the normal JPEG compression process is resumed 408-2. If candidate regions are 
determined 908 then a bounding region is determined for each 910 which is sufficiently large 
to include various eye -region features which may be used as part of the main prior-art redeye 
filter process 406 of Fig 3(a). If the certainty of the region being a flash eye defect is high 

20 enough, a bounding box region is decompressed to bitmap format 912 and a speed optimised 
redeye filter chain 914 is applied to correct that region of the main image 914. The corrected 
regions in bitmap space are next mapped to an integer number of 8x8 block boundaries and 
are recompressed 918 and subsequently overwritten 920 onto the DCT domain. Finally, 
normal JPEG compression is resumed 408-2. As mentioned previously each of the corrected 

25 region boundaries and suspected region boundaries are output for use in later analysis 
optimized detection and correction. 

Fig 3(c) shows the region based analysis 907 of Fig 3(b) in more detail. Firstly, the 
DCT coefficients are read 930 from a DCT image in temporary memory store. These 
coefficients are then preprocessed into a set of criteria tables 932. Each table is essentially a 

30 numeric table of size NxM where there are NxM DCT blocks in the image being analyzed. As 
examples, one such table will contain the red chrominance component normalized to 
emphasize a colour range associated with flash eye defects and derived from the DC 
coefficients for the luminance (Y) and red chrominance (Cr) components of each DCT block. 
Another table may contain differential values derived from neighbouring DCT blocks and 
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used in edge detection; yet another table may contain variance values calculated across a set 
of neighbouring DCT blocks. Those skilled in the art will realize that as an implementation of 
the DCT prefilter becomes increasingly sophisticated that multiple additional criteria may be 
incorporated into the algorithm. 
5 After the calculations required for each criteria table have been completed 932 they 

are copied into temporary storage 933 and the prefilter algorithm will next perform a filtering 
and segmentation step 907 for each of the plurality of criteria tables. This particular step is 
further detailed in Fig 3(d) below. Now the prefilter has determined a plurality of sets of DCT 
block grouping based on the segmentation analysis of a plurality of criteria tables. It is now 
1 0 necessary to sort and analyze these grouping to determine a final set of flash defect candidate 
regions. 

This region-based analysis 936 is comprised of a number of alternative techniques 
which will be known to those skilled in the art. In particular, we mention that regions may be 
combined both in inclusive, exclusive and less frequently in mutually exclusive combinations 
15 936-1; an alternative approach to region-based analysis will employ template matching 936-2, 
one example of which is disclosed US 5,805,727 to Nakano discloses matching a subregion 
within a DCT image using both coarse and fine template matching techniques based on the 
DC coefficients of the DCT blocks within the image. 

An important component of the region based analysis is a re-segmentation engine 92-6 
20 which is responsible for analyzing larger regions which may, in fact, be two distinct 
overlapping regions, or clusters of smaller regions which may, in fact, be a single larger 
region. Then once the region based analysis 936 is completed a final LUT containing the list 
of determined flash defect candidate regions is obtained and written to system memory. 

Fig 3(d) shows the segmentation step 907 of the redeye DCT prefilter in more detail. 
25 The next preprocessed criteria table to be processed by the segmentation process is first 
loaded 950 and the labeling LUT for the region grouping process is initialized 952. Next the 
current DCT block and DCT block neighbourhoods are initialized 954. 

Fig 3(e) shows a diagrammatic representation of a 4-DCT block neighborhood 992, 
shaded light grey in the figure and containing the three upper DCT blocks and the DCT block 
30 to the left of the current DCT block 994, shaded dark grey in the figure. This 4-block 
neighborhood is used in the labeling algorithm of this exemplary embodiment. A look-up 
table, LUT, is defined to hold correspondence labels. 

Returning to step 954 we see that after initialization is completed the next step for the 
workflow of Fig 3(d) is to begin a recursive iteration through all the elements of the current 
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criteria table in a raster-scan from top-left to bottom-right The workflow next determines if 
the current criteria table value, associated with the current DCT block satisfies membership 
criteria for a candidate redeye region 958. Essentially this implies that the current criteria 
table value has properties which are compatible with a flash eye defect. If the current criteria 
5 table value satisfies membership criteria for a segment 958, then the algorithm checks for 
other member DCT blocks in the 4-block neighborhood 960. If there are no other member 
blocks, then the current block is assigned membership of the current label 980. The LUT is 
then updated 982 and the current label value is incremented 984. If there are other member 
blocks in the 4-block neighborhood 960 then the current block is given membership in the 

10 segment with the lowest label value 962 and the LUT is updated accordingly 516. After the 
current block has been labeled as part of a flash-eye defect segment 962 or 980, or has been 
categorized as not being a member of a candidate defect region during step 958, a test is then 
performed to determine if it is the last DCT block in the image 966. If the current block is the 
last block in the image then a final update of the LUT is performed 970. Otherwise the next 

15 criteria table value is obtained by incrementing the current block pointer 968 and returning to 
step 958 and is processed in the same manner. Once the final DCT block is processed and the 
final LUT completed 970, all of the blocks with segment membership are sorted into a 
labeled-segment table of potential eye-defect segments 972. Another test is then made to 
determine if this is the last criteria table to be processed 966 and if that is the case then 

20 control is passed to the region based analysis step of Fig 3(c) 936. Otherwise the block 
segmentor returns to step 950 and loads the next criteria table for processing. 

A number of conventional techniques can advantageously be adapted for use within 
the redeye DCT prefilter. US 5,949,904 to Delp discloses querying image colors within a 
DCT block. In particular it allows the determination of colour within the DCT block from the 

25 DC coefficient of the DCT alone. Thus from a knowledge of the DC coefficients alone color 
matching can be achieved. US 6,621,867 to Sazzad et al discloses determining the presence 
of edges within DCT blocks based on differences between the DC coefficients in 
neighbouring DCT blocks. 

Now additional image qualities such as texture and image sharpness/blur can be 

30 determined through an analysis of additional AC components within a DCT block. Examples 
of such analysis techniques are described in US patent application No. 2004/0120598 to Feng 
and US patent application No. 2004/0057623 to Schuhurke et al. 

Alternative DCT block segmentation techniques may be employed in other 
embodiments, specifically techniques described in prior art patents US 6,407,777 to DeLuca, 
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US 6,873,743 to Steinberg, and US patent applications 2005/0047655 and 2005/0047656 to 
Luo et al may be advantageously employed in certain alternative embodiments. 

In Fig 4 we show an example of how an outline colour template can be constructed for 
redeye regions. Fig 4(a) shows an eye regions mapped onto a rectangular grid. Each block of 
5 the grid 201 corresponds to an 8x8 pixel block. The main redeye defect 204 is typically 
surrounded by an iris region 203 and an additional eye-white region 202 and the boundary of 
the main redeye region, 206 as determined by a conventional redeye filter. 

Next, in Fig 4(b) we show the approximate colour which will be recorded by the DC 
coefficient of each DCT block after the image in Fig 4(a) is transformed into the DCT 

10 domain. The colour combinations shown in Fig 4(b) are as follows: R is a reddish hue 
indicative of a flash-eye defect phenomenon; S is a hue indicative of a skin colour; W: 
indicates a whitish colour associated with the eye-white region; I: is the Iris colour of the eye 
which can vary significantly from person to person; WS: indicates a block with mixed skin 
and eye-white; RW: is a block with mixed redeye and eye white; and RI: has a hue which is a 

15 mix of red and the Iris colour. Now if sufficiently inclusive colour filtering is applied to these 
image blocks it is possible to determine directly from the DC coefficients of the DCT domain 
image a colour map for a typical redeye. In Fig 4(c) we illustrate the region which will be 
determined as red if an inclusive colour filter is used. Figs 4(d) and 4(e) illustrate this for eye 
white and skin colour regions surrounding the flash eye defect region. This data can, for 

20 example, be used to build a set of colour templates for a flash eye defect region. By applying 
other conventional techniques it is possible to determine DCT blocks which contain sharp 
edges, or transitions between colour regions. This can provide additional information to 
improve the DCT prefilter detection process. 

A potential disadvantage in the embodiment of Fig 3(a) is that it requires the entire 

25 image to be decompressed in order to perform the second-step redeye filtering process. As 
most cameras use JPEG compression which is lossy it is desirable for certain embodiments to 
implement a lossless embodiment which allows a two-stage redeye process to be applied 
within an image acquisition appliance without loss of image quality. 

Accordingly we present in Fig 5 a functional implementation of modified redeye 

30 filtering process which allows an analysis optimized redeye detection and correction to occur 
in playback mode, without loss of image quality. This also allows complex post-processing, 
to be implemented in incremental steps. Thus, when a camera is idle with respect to user 
activity, yet is still switched on it may load and commence processing of an image. When 
user activity recommences the camera can recompress and save the image being processed 
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prior to responding to the user. As the embodiment described below allows lossless saving 
and restoration of a image within the camera, it thus facilitates incremental process of an 
image which is not limited to redeye, but may be applied likewise to other in-camera methods 
such as face detection or recognition. 
5 Various means of sensing user activity will be known to those skilled in the art. One 

exemplary means include detecting camera motion and optionally correlating this with other 
in-camera functions such as the autofocus subsystem and the user-interface subsystem. Many 
cameras also incorporate a power-saving mode which determines that a camera has been 
inactive long enough to disable certain main subsystem. When such a mode is activated by 

10 user inactivity then additional background image processing can be initiated without 
interfering with the use of the appliance by the user. 

Returning to Fig 5 we illustrate an embodiment of the present invention which 
incorporates a speed-optimized redeye filter 41 1 in the main image acquisition chain 401. In 
this exemplary embodiment the speed optimization of the filter is achieved by implementing a 

15 minimal set of falsing and validation filters and no correction process is applied during the 
main image acquisition chain. In alternative embodiments the speed optimization techniques 
described in relation to embodiments above may optionally be incorporated or substituted. 

After an image is analyzed by this speed optimized redeye filter 411 it is subsequently 
compressed 427-1 and stored 410. In addition data relating to the location of candidate redeye 

20 regions and false positives is recorded and associated with the stored image. 

Now when the camera can initiate background processing, as described above, or 
when the user enters playback mode and selects an image for viewing it will be partially 
decompressed 433 from JPEG to DCT block form. As this decompression step is lossless 
there is no loss of quality to the main image which is temporarily stored in memory and 

25 passed to a DCT region decompressor 430. This DCT region decompressor uses the data 
stored and associated with the original image to determine the specific DCT blocks which 
contain candidate redeye regions, and, optionally, false positive regions which may benefit 
from additional detection processing if sufficient time & system resources are available. 

Each decompressed DCT region is then incrementally filtered by one or more redeye 

30 filters to determine corrections which should be applied to said DCT image block. 

In certain embodiments, DCT blocks may be decompressed to bitmap format and 
filtered as a pixel block. In other embodiments adjacent, non-candidate DCT blocks may be 
included in the decompression 430 and filtering 412 processes. Once a decompressed DCT 
block region, which may consist of multiple DCT blocks, has been corrected by the redeye 
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filter 412 then the corrected DCT image segment is passed onto a DCT block matching 
module 416 which, in addition to checking the alignment of DCT blocks will also check the 
integration of the corrected DCT blocks within the partially decompressed and temporarily 
stored DCT block image. When all candidate DCT blocks and any adjacent DCT blocks 
5 included in the redeye filter analysis have been corrected they are overwritten onto the 
partially decompressed and temporarily stored DCT block image by a DCT region overwriter 
418 module. The partially decompressed and temporarily stored DCT block image is next 
passed to the DCT to JPEG image compression module 427-1 and is losslessly compressed 
back to JPEG format. 

10 Note that in this way the only regions of the image which are decompressed using 

lossy techniques are those identified by the speed optimized redeye filter 4 1 1 in the image 
acquisition chain. As these image regions are to be corrected the effect of lossy 
decompression and recompression on them will thus be negligible. 

Several alternative embodiments of the invention can be identified. These include (i) 

15 saving a copy of the original defect region prior to overwriting the DCT blocks which contain 
the image defect in the temporary copy of the DCT image. This alternative embodiment 
supports lossless restoration of the original image. Those skilled in the art will realize that the 
saved original DCT block region containing the defect can be stored within the header of the 
JPEG image. One prior art description of such a technique is provided in US 6,298,166 to 

20 Ratnakar et al. where watermark data is incorporated in the image. Thus the corrected image 
can contain a copy of any original uncorrected regions; alternatively (ii) multiple alternative 
correction algorithms can be employed and these may be temporarily copied for later storage 
in the JPEG header for later selection by an end user through a user interface, either on the 
camera or subsequently in a computer based image processing application. Note that in this 

25 the overwriting step is optional; if it is used then certain image analysis criteria can be applied 
as an additional processing step either immediately prior to overwriting, or as an integral part 
of detecting or correcting red-eye or combinations thereof. 

Further aspects of these embodiments are illustrated in Fig 6(a)-(c) which show 
respectively: Fig 6(a) an example of the original defect region 506 stored in the header 504 

30 and the corrected defect region 508 applied to the main image body 502; Fig 6(b) an example 
of the corrected defect region 508 stored in the header 504 and the original defect region 506 
remaining uncorrected in the main image body 502; Fig 6(c) an example of the original defect 
region 506 and at least one alternative corrected defect region 508-2 stored in the header 504 
and the optimally determined corrected defect region 508-1 applied to the main image body 
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502. Note that the graphical representations of "corrected" and "uncorrected" eye regions 
used in Fig 6 is for illustrative purposes only; those skilled in the art will realize that each 
graphical eye-region actually represents a transformed block of DCT coefficients. 

In other embodiments, the performance of the fast red-eye filter can be further 
5 improved by selectively applying it to a limited set of regions within the acquired image. As it 
is generally impractical to implement extensive image analysis during the main image 
acquisition chain, these regions are preferably determined prior to the initiation of the main 
image acquisition. 

One convenient approach to pre-determine image regions which have a high 

10 probability of containing red-eye candidates is to perform pre-processing on a set of preview 
images. A digital camera can generally include components for enabled the acquisition of 
such a stream of images, e.g., captured at video rates of 15-30 frames per second (fps) at a 
lower resolution than that provided by the main image acquisition. A set of 320x240, or 
QVGA images is typical of many consumer cameras and the size and frame-rate of this 

15 preview images stream can normally be adjusted within certain limits. 

In two exemplary embodiments, as illustrated in Figures 7 and 8, the digital camera 
includes a face detector (600) which operates on the preview image stream (410-3). Figure 7 
includes a face detector 600, while Figure 8 illustrates a face detector and tracker 600, which 
includes tracking of detected faces across multiple frames. Figure 8 also illustrates a preview 

20 stream acquisition sub-system 620, including image sensor subsystem 610, and a display 605, 
as well as the face detector and tracker 600 itself. 

Face detecting and tracking typically involve two principle modes: (i) a full image 
search mode to detect (and confirm) new face-candidate regions (601) and (ii) the main 
tracking mode which predicts and then confirms the new location of existing face- candidates 

25 in subsequent frames of the image stream and compiles statistical information relating to each 
such confirmed candidate region. Both modes can employ a variety of new and/or 
conventional methods including face detection, skin region segmentation, feature detection 
including eye and mouth regions, active contour analysis and even non-image based inputs 
such as directional voice analysis (e.g. US 2005/0147278 to Rui et al which describes a 

30 system for Automatic detection and tracking of multiple individuals using multiple cues). As 
the first mode, hereafter referred to as the "seeding mode" is applied to the entire image it is 
computationally more intensive and is only applied occasionally - typically every 30-60 
image frames. As such, new faces appearing in the image will still be detected within a 
couple of seconds which is sufficient for most consumer applications. The second mode is 
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preferably applied to every image frame, although not all of the analysis cues may be applied 
on every frame. 

Thus in normal operation only the output(s) from the second operational mode of a 
face tracker algorithm will be available after every frame of the preview image stream. There 
5 are three principle outputs from this second mode: (i) a list of candidate face regions which 
are confirmed to still contain faces; and/or (ii) a set of data associated with each such 
confirmed face region including its location within that frame of the image and various 
additional data determined from a statistical analysis of the history of said confirmed face 
region; and/or (iii) a predicted location for each such confirmed face region in the next frame 

10 of the preview image stream. Note that if item (ii) is used, item (iii) can be optional as 
sufficient data may be provided by item (ii) for a determination of predicted location. 

Now it can be understood that in our embodiment these outputs from the preview face 
detector (600) enable the speed optimized red-eye detector 411 to be applied selectively to 
face regions (601) where it is expected that a red-eye defect will be found. 

15 A face detector may advantageously be first applied to an image prior to the 

application of a red-eye filter (see, e.g., US 20020172419 to Lin et al; US 20020126893 to 
Held et al; US 20050232490 to Itagaki et al and US 20040037460 to Luo et al., which are 
incorporated by reference). Under normal circumstances, there generally is not sufficient time 
available during the main image acquisition chain, which is operable within a digital camera, 

20 to allow the application of the face detector prior to the application of a red-eye filter. The 
present embodiment overcomes this disadvantage of the prior art by employing the predictive 
output of a face tracker module (600). Although the size of the predicted region will typically 
be larger than the size of the corresponding face region it is still significantly smaller than the 
size of the entire image. Thus the advantages of faster and more accurate detection can be 

25 achieved within a digital camera or embedded image acquisition system without the need to 
operate a face detector (600) within the main image acquisition chain. 

Note that where multiple face candidate regions (601) are tracked, then in certain 
embodiments, multiple predicted regions will have the speed-optimized red-eye filter applied. 
Referring still to Figures 7 and 8, a main image may be acquired, subsampled and 

30 stored before being processed by a face detector/tracker 600 as in Figure 7, or a face 
detector/tracker 600 may be applied prior to storage as in Figure 8 and perhaps in parallel 
with main image acquisition. A separate "preview stream" of (uncompressed) images may be 
piped from the main image sensor, independent of the main acquisition, as illustrated at 
Figure 8. The preview stream may be sent to the camera display 605. It is this preview 
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stream that the face tracker 600 may be operable on and from whence the candidate face 
regions may be drawn in the embodiment of Figure 8. In using the face tracker 600, a 
predicted location of a face is obtained from a preview image, and the speed optimized red- 
eye filter may then be operable on the main acquired image (or a subsampled copy thereof). 
5 While an exemplary drawings and specific embodiments of the present invention have 

been described and illustrated, it is to be understood that that the scope of the present 
invention is not to be limited to the particular embodiments discussed. Thus, the 
embodiments shall be regarded as illustrative rather than restrictive, and it should be 
understood that variations may be made in those embodiments by workers skilled in the arts 

10 without departing from the scope of the present invention. 

In addition, in methods that may be performed according to preferred embodiments 
herein and that may have been described above, the operations have been described in 
selected typographical sequences. However, the sequences have been selected and so ordered 
for typographical convenience and are not intended to imply any particular order for 

15 performing the operations, except for those where a particular order may be expressly set 
forth or where those of ordinary skill in the art may deem a particular order to be necessary. 

In addition, all references cited herein as well as the background, invention summary, 
abstract and brief description of the drawings, as well as US patent no. 6,407,777, 7,315,631 
and 7,336,821, and US published patent applications nos. 2005/0041121, 2005/0031224, 

20 2005/0140801, 2006/02041 10, 2006/0093212, 2006/0120599, 2007/01 10305 and 
2006/0140455, and PCT/EP2006/008358, and US patent applications nos. 60/773,714, 
60/804,546, 60/865,375, 60/865,622, 60/829,127, 60/829,127, 60/821,165, 60/892,882, 
60/945,558, 60/915,669 10/772,767, 11/554,539, 11/464,083, 11/462,035, 11/282,954, 
11/027,001, 10/842,244, 11/024,046, 11/233,513, 11/753,098, 11/753,397, 11/766,674, 

25 11/769,206, 11/772,427 and 11/460,218, are all incorporated by reference into the detailed 
description of the preferred embodiments as disclosing alternative embodiments. 
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What is claimed is 

1 . A method of detecting a red-eye defect within a digital image 

(a) acquiring an image including one or more faces including a red eye defect; 
5 (b) performing in a first stage an initial segmentation of candidate redeye regions to 

determine a first set of one or more confirmed redeye regions designated for correction; 

(c) correcting the first set of red eye regions for generating a first red eye corrected 

image; 

(d) recording a subset of rejected candidate red eye regions that were rejected for 
10 correction in the first stage without full analysis; 

(e) re-analyzing the subset of rejected candidate red eye regions in a later second stage 
to determine a second set of confirmed redeye regions; 

(f) correcting the second set of confirmed red eye regions and generating a second red 
eye corrected image which has the first and second sets of confirmed red eye regions 

1 5 corrected therein; and 

(g) electronically storing, transmitting, further processing or editing, or displaying the 
second red eye corrected image, or combinations thereof. 

2. The method of claim 1, further comprising applying a speed- optimized set of one or more 
20 falsing/verification filters in the first stage. 

3. The method of claim 1, further comprising re-analyzing the first set of confirmed red eye 
regions in the second stage. 

25 4. The method of claim 1, wherein the initial segmentation is applied upon acquisition of the 
image and the second later stage occurs in a background or playback mode or both. 

5. The method of claim 1, further comprising displaying in real time the first red eye 
corrected image on a display screen prior to the later second stage without input from a user. 

30 

6. An embedded image acquisition and processing system, comprising. 

(a) an image acquisition subsystem; 
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(b) a speed-optimized filter that performs in a first stage an initial segmentation of 
candidate redeye regions detected within an acquired image to determine a first set of one or 
more confirmed redeye regions designated for correction; 

(c) a storage device to record a subset of rejected candidate red eye regions that were 
5 rejected for correction in the first stage without full analysis; 

(d) an analysis-optimized filter that determines in a later second stage a second set of 
one or more confirmed red eye regions designated for correction; 

(e) a processor for correcting the red eye defect including the first and second sets of 
confirmed red eye regions within the acquired image and generating a red eye corrected 

10 image; and 

(f) wherein the red eye corrected image is electronically stored, transmitted, further 
processed or edited, or displayed, or combinations thereof. 

7. The system of claim 6, wherein the speed-optimized filter applies a speed-optimized set of 
15 one or more falsing/verification filters. 

8. The system of claim 6, wherein the first set of confirmed redeye regions is also passed 
through the analysis -optimized filter. 

20 9. The system of claim 8, wherein the analysis-optimized filter includes an enhanced 
correction filter for application to the first set of one or more confirmed red eye regions. 

10. The system of claim 6, wherein the speed-optimized filter is applied upon acquisition of 
the image and the analysis-optimized filter is applied in a background or playback mode or 

25 both. 

1 1 . The system of claim 6, further comprising a "RE lib" component for displaying in real 
time an initial red eye corrected image on a display screen prior to the later second stage 
without input from a user. 

30 

12 . A method of detecting eye defects within digital images, comprising: 

(a) acquiring an image including one or more faces including red and non-red eye 

defects; 
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(b) performing in a first stage an initial segmentation of candidate redeye regions to 
determine a first set of one or more confirmed redeye regions designated for correction; 

(c) correcting the first set of red eye regions for generating a first red eye corrected 

image; 

5 (d) performing in a later second stage a segmentation of candidate golden eye or white 

eye regions, or both, to determine a second set of one or more confirmed non-red eye defect 
regions also designated for correction; 

(e) correcting the second set of non-red eye defect regions within the first red eye 
corrected image for generating an red and non-red eye defect corrected image; 
10 (f) electronically storing, transmitting, further processing or editing, or displaying the 

second red eye corrected image, or combinations thereof. 

13. The method of claim 12, further comprising applying a speed-optimized set of one or 
more falsing/ verification filters in the first stage. 

15 

14. The method of claim 12, further comprising re-analyzing the first set of confirmed red 
eye regions in the second stage. 

15. The method of claim 12, wherein the initial segmentation is applied upon acquisition of 
20 the image and the second later stage occurs in a background or playback mode or both. 

16. The method of claim 12, further comprising displaying in real time the first red eye 
corrected image on a display screen prior to the later second stage without input from a user. 

25 17. An embedded image acquisition and processing system, comprising. 

(a) an image acquisition subsystem; 

(b) a speed-optimized filter that performs in a first stage an initial segmentation of 
candidate redeye regions detected within an acquired image to determine a first set of one or 
more confirmed redeye regions designated for correction; 

30 (c) an analysis- optimized filter that determines in a later second stage a second set of 

one or more confirmed non-red eye regions designated for correction; 

(d) one or more processors for correcting the red and non-red eye defects including the 
first and second sets of confirmed red and non-red eye regions, respectively, within the 
acquired image and generating a red and non-red eye corrected image; and 
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(e) wherein the red and non-red eye corrected image is electronically stored, 
transmitted, farther processed or edited, or displayed, or combinations thereof. 

18. The system of claim 17, wherein the speed-optimized filter applies a speed-optimized set 
5 of one or more falsing/verification filters. 

19. The system of claim 17, wherein the first set of confirmed redeye regions is also passed 
through the analysis -optimized filter. 

10 20. The system of claim 19, wherein the analysis-optimized filter includes an enhanced 
correction filter for application to the first set of one or more confirmed red eye regions. 

21. The system of claim 17, wherein the speed-optimized filter is applied upon acquisition of 
the image and the analysis-optimized filter is applied in a background or playback mode or 

15 both. 

22. The system of claim 17, further comprising a "RE lib" component for displaying in real 
time an initial red eye corrected image on a display screen prior to the later second stage 
without input from a user. 

20 

23. A method of processing a digital image comprising: 

(a) obtaining candidate face regions of an acquired digital image; 

(b) filtering the candidate face regions with a first speed optimized filter to produce a 
first set of candidate red-eye regions; 

25 (c) encoding at least a portion of the acquired digital image; and 

(d) storing the encoded portion of the acquired digital image in association with said 
first set of candidate red-eye regions for later image processing of said encoded image. 

24. The method of claim 23, further comprising applying a speed- optimized set of one or 
30 more falsing/verification filters. 

25. The method of claim 23, further comprising passing the first set of candidate redeye 
regions through an analysis -optimized filter. 
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26. The method of claim 23, farther comprising passing candidate face regions rejected by 
the filtering through an analysis -optimized filter sometime after the filtering, encoding and 
storing. 

5 27. The method of claim 26, wherein the first speed optimized filter is applied upon image 
acquisition, and the analysis- optimized filter is applied in a background or playback mode or 
both. 

28. A system for processing a digital image, comprising: 
10 (a) an image acquisition subsystem; 

(b) a face detector for obtaining candidate face regions of an acquired digital image; 

(c) a speed optimized filter for filtering the candidate face regions to produce a first 
set of candidate red-eye regions; 

(d) a processor for encoding at least a portion of the acquired digital image; and 

15 (e) storing the encoded portion of the acquired digital image in association with said 

first set of candidate red-eye regions for later image processing of said encoded image. 

29. The system of claim 28, further comprising an analysis-optimized filter for filtering at 
least initially rejected candidate face regions to produce a second set of candidate red-eye 

20 regions. 

30. The system of claim 28, further comprising an analysis-optimized filter for confirming 
one or more of the first set of candidate red eye regions sometime after speed-optimized 
filtering. 

25 

31. The system of claim 30, wherein the first speed optimized filter is applied upon image 
acquisition, and the analysis- optimized filter is applied in a background or playback mode or 
both. 
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